PROTOTYPE: move L2G training and prediction from gentropy into PTS - #106

Nine TDD tasks: the feature contract, gold-standard parsing, the pinned
split derivation, model fit/evaluate/persist, the SHAP process pool, the
two transformer entry points with their config step, the orchestration
retirement of the three gentropy steps, and a gated fixed-model parity
harness against a published release.

Also corrects the design's config sketch: the feature list is shared
between the two tasks by a YAML anchor, not a scratchpad sentinel, because
otter's Scratchpad is string.Template substitution and would deliver a
list as its repr.

Seven findings from the pre-execution conflict scan, ruled and applied:

- impute before cast reversed to cast before impute, matching gentropy's
  L2GFeatureMatrix, so the parity gate is not undermined by a rounding order
- a Task 6 test asserted nothing (hasattr(pl, 'Path') is False, so the
  ternary always evaluated True); now asserts the artifact exists
- pts_vep_view also depends on gentropy_l2g_prediction, which Task 8 never
  mentioned; repointing it to pts_l2g, without which the DAG cannot build
- the empty-prediction test now pins whatever write_dataset does with a
  zero-row frame instead of assuming a part file exists
- shapValue is null rather than NaN when explanations are off
- a config test guards the YAML anchor against yamlfmt expanding it and
  letting the two tasks' feature lists drift
- numpy imported normally rather than via __import__

xgboost>=3.4.1 and shap>=0.52.0 were an accident of the scratch benchmark
venv, built on Python 3.12. uv forks both by Python version, so those
floors would have forced pts off Python 3.11 and off its python:3.11-slim
production base image for no functional gain.

The real obstacle is narrower and unrelated to the Python floor: on x86_64
macOS only, shap requires numba<0.63, numba caps numpy<2.4, and pts
requires numpy>=2.4.4. Excluding that single platform via tool.uv
environments is the smallest fix, and unlike a linux+arm64 allowlist it
keeps the Windows-only packages in the lock.

Add xgboost, shap, scikit-learn, and skops as pts dependencies, and the
shared L2G feature contract (FEATURES, FIXED_COLUMNS, LOCUS_MEAN_IMPUTED,
impute_and_cast) that later L2G training/prediction tasks build on.

xgboost and shap are floored at 3.2.0/0.51.0 rather than 3.4.1/0.52.0:
those newer floors only resolve on Python >=3.12, and pts stays on
>=3.11 (production is python:3.11-slim). uv forks the two ranges by
Python version, so this only changes what a >=3.12 dev environment
would resolve to; production gets xgboost 3.2.0 / shap 0.51.0.

Also add [tool.uv] environments, narrowed to exclude x86_64 macOS:
shap's numba<0.63 pin collides with the baseline_expression extra's
numpy>=2.4.4 requirement on that platform only, and is otherwise
unsatisfiable. pts can no longer be locked/installed on an Intel Mac;
production (linux) and local dev (Apple Silicon) are unaffected.

The 317 rows/s in the design was measured on an idle machine. Re-measured
under concurrent load, on the same rows and the same background, it is 181
rows/s -- so the design overstated by ~1.7x and the table now carries the
range with the slow end as the sizing basis.

A controlled same-rows comparison isolates the library versions at ~6%
(181 against 193 rows/s), so pinning production to xgboost 3.2.0 and shap
0.51.0 costs almost nothing. That pair also loads the released model and
reproduces its published scores at max |delta| = 0.

Conclusion is unchanged: under 5 vCPU-hours against the Batch job's
envelope of up to 2,000, so a single VM remains the right shape.

Also records that chunks must outnumber workers -- a chunk count below the
worker count caps the speedup regardless of core count.

Per review: a per-file # isort: skip in model.py would have to be repeated in
every future l2g module that touches both xgboost and skops, and a missed
file reopens a SIGSEGV that names neither library. Import xgboost once in the
package __init__ instead, so every submodule inherits the ordering.

The l2g/__init__.py guard only wins the race once that package is imported;
a test module importing skops before ever touching pts.transformers.l2g
still loses it. Import xgboost in tests/conftest.py, which pytest always
loads before any test module in the directory, so the ordering holds
regardless of collection order.

Verified with a throwaway repro test (skops imported before the l2g
package): segfaults without this guard, passes with it.

…test

evaluate() computed averagePrecision from predict_proba scores; gentropy's
L2GTrainer.evaluate computes it from the hard 0/1 predictions instead. The
two differ by ~0.15 on the real 26.09-2 test split (0.781 vs 0.934), so a
silent formula change would misread as a large model improvement against
recorded runs. Report both: averagePrecision now matches gentropy exactly,
and averagePrecisionFromScores carries the conventional (scores-based) form.

Also strengthen test_fit_is_deterministic_for_a_fixed_seed: assert the
fitted model actually reports random_state=777, since the prior bitwise-
equality check alone would still pass even if random_state were dropped
from DEFAULT_HYPERPARAMETERS.

Add a regression test that runs l2g_train twice (train_on_full_dataset True/False)
on a fixture designed so a leak between the refit and evaluate() is unmissable --
the original shared fixture put every positive on genes the pinned test set also
marked positive, so contamination stripped all positives out of training and made
held-out metrics identical regardless of ordering. The new fixture gives train and
test disjoint gene pools so training keeps real positives, and a leaked refit
memorises the test rows outright (every metric hits 1.0) against an honest
~0.5-0.75 baseline. Also assert the two runs save different models, so the flag
is not proven safe by being inert.

Add test_config_l2g.py asserting config.yaml's l2g_train hyperparameters equal
model.DEFAULT_HYPERPARAMETERS exactly, guarding the hand-duplicated block the
same way the &l2g_features anchor guards the feature list.

Retires gentropy_l2g_train_test_split, gentropy_l2g_training and
gentropy_l2g_prediction, including the 1000-task Google Batch job and its
l2g_predict.sh entrypoint. gentropy_l2g_evidence now depends on pts_l2g.

In production `release_uri` is set, so otter resolves every relative destination in config.yaml
into a `gs://…` URI before a transformer sees it. Three of l2g_train's writers were pathlib- or
skops-backed, and POSIX collapses `gs://` to `gs:/`: the model, metrics.json and the split
statistics landed under the container's working directory and the release shipped without them,
silently, because l2g_predict then read the model back from the same collapsed local path.

- `_write_json` and `save_model`/`load_model` route through otter's StorageHandle.
- The background parquet drops its `Path(...).parent.mkdir` for `write_parquet(mkdir=True)`, the
  idiom release_metrics already uses for a single non-dataset release artifact.

Also in this wave:

- The gold standard's own `studyLocusId` is dropped before the join, as gentropy does. Kept, it
  arrived as `studyLocusId_right` and the dedup could not see past it, so two curated rows
  differing only in credible-set assignment became two training rows. Latent on today's curation
  file; the existing test could not see it because its two gold rows shared a studyLocusId.
- The SHAP pool pins the `spawn` start method. The default is spawn on macOS but fork on Linux,
  and production is python:3.11-slim; forking after predict_proba and TreeExplainer have run
  their OpenMP regions is the classic hang, with no execution_timeout to bound it. Every test of
  this path has run under spawn, so this makes production run the exercised configuration.
- The parity gate calls l2g_predict instead of re-implementing its preparation. Deleting the
  isProteinCoding filter, or changing the threshold comparison, previously left it passing.
- OUTPUT_COLUMNS is enforced by a final select rather than merely documented; the config
  features_list is checked against FEATURES, not only against its own copy; and three docstrings
  quoting superseded measurements are corrected against the design spec.

l2g_training was the only consumer of wandb-key and it has moved into pts, which does not use
Weights & Biases. Nothing else references W&B, so every gentropy cluster was fetching a secret no
step reads -- and would have failed to create at all once the secret was retired.

HF_TOKEN stays: hfhub-key is still fetched by operators/gce.py.

…pread

The full-scale gate against do/platform-2609-1, using that release's own
model over its own feature matrix across all 200 partitions, reproduced
every one of the 3,219,816 published rows with max |delta| 1.192e-07 and
none above the 1e-6 tolerance, and matched the threshold row set exactly.

Corrects a third sampled figure. The design said shapBaseValue varies
1.75x across partitions, from six sampled parts. Across all 200 there are
200 distinct values spanning 0.028042 to 0.137713, a 4.91x spread. The
argument for a single seeded background is stronger than the design
claimed, not weaker.

…annot see

The rewritten parity gate catches a deleted isProteinCoding filter, but it cannot catch the
threshold comparison being flipped from `>=` to `>`: no released row scores exactly 0.05 in
float64, so both produce identical output and no gate over released data could tell them apart.
The module docstring implied otherwise.

- The parity docstring now states precisely which mutations the gate catches and which it cannot,
  and points at the unit test that covers the boundary.
- That unit test derives its threshold from a score the model actually produced -- score
  everything, take a real score, re-run with it as the threshold -- so the boundary row and the
  threshold are the same number and no floating-point engineering is needed. Verified to fail
  under `>`: "the row scoring exactly at the threshold was dropped".

polars joins leave their output order unspecified, and `derive_splits` ends in
one, so the same inputs came back in a different row order on every run --
three orders from three calls in a single process. Both things downstream are
positional: XGBoost's `subsample=0.8` draws its rows by position, and
`build_background`'s `rng.choice` draws the SHAP background by position. The
fitted model and `shapBaseValue` therefore differed between runs while
`metrics.json` recorded `shapBackgroundSeed` as though the draw were
reproducible.

Sort train and test on `(studyLocusId, geneId)` -- the natural key, unique in
this data -- immediately after the split, which is where the positional
dependency begins. `labelled` inherits the order through `pl.concat`.

The new test runs `l2g_train` twice from one fixture and requires the two saved
models to score identically and the two `shap_background.parquet` files to be
equal. Without the sort it fails on both halves: the backgrounds differ, and
the models score the test split 0.03907016 against 0.04322654.

`shap_background_size` was set twice, once under `transform l2g_train` and
again under `transform l2g_predict`, with nothing binding the two. Raise the
training one to 1000 and forget the other and `Independent(max_samples=100)`
would silently use 100 of the 1000 rows while `metrics.json` recorded 1000 --
the same truncated-background defect this migration exists to remove.
`features_list` is held together by a YAML anchor and `hyperparameters` by a
test, and this had neither.

The masker is now sized as `max_samples=background.shape[0]`, from the parquet
training wrote, and the setting is gone from the predict task. A config test
asserts it stays gone.

Also from the review:

* the `OUTPUT_COLUMNS` re-select claimed a reordering would raise. It is
  silently corrected instead, and only a rename or a drop raises. Reworded.
* `build_output` publishes the imputed feature value where gentropy re-joins
  the raw matrix and would publish a null. Recorded as deliberate -- the value
  should be the one its `shapValue` explains -- and unreachable here, since all
  31 features are null-free across all 200 partitions of the real matrix.
* `test_l2g_predict_output_is_sorted_by_key` asserted only `studyLocusId` on a
  fixture with one gene per locus, so sorting by that alone would have passed.
  It now runs on twelve loci of four genes each, staged in descending key
  order, and compares the key tuples.
* `test_l2g_predict_applies_the_threshold` accepted either outcome behind
  `if parts:`. The behaviour is determinate, `pl.PartitionBy` writing one
  readable zero-row `00000000.parquet`, so assert that and drop the branch.

Three documents contradicted the shipped code.

* The design's training walkthrough said `evaluate` reports "the same six
  metrics gentropy reports". It reports seven. `averagePrecision` keeps
  gentropy's hard-label formula so earlier runs stay comparable, and
  `averagePrecisionFromScores` is the conventional form. They differ by roughly
  0.15 on the real test split, so the two are now named and distinguished.
* `explain.py` and `test_l2g_parity.py` quoted the published `shapBaseValue`
  spread as 0.0381 to 0.0668, a 1.75x range from a six-partition sample. The
  design already retracted it. Measured across all 200 partitions of
  `do/platform-2609-1` it is 200 distinct values from 0.028042 to 0.137713, a
  4.91x spread. The design's own acceptance section still repeated the old
  figure and now agrees with itself.

The parity test's `> 1.5` assertion is left alone. It still holds and a tighter
one would be calibrated to a single release, which is now said in place.

`test_features_has_31_names_in_fitted_order` checked the count, the uniqueness
and the two ends, so any transposition among the 29 names between them passed.
It now asserts the whole tuple against an explicit literal.

`CURATED_COLUMNS` omits `traitFromSourceMappedId`, which gentropy's
`L2GGoldStandard` schema allows. A future curated file carrying it would have
it dropped, and two rows differing only there would collapse in `annotate`'s
`unique`. Documented, along with why gentropy's
`n_unique_positive_gene_disease_pairs` statistic is absent for the same reason.
Behaviour unchanged.

It was stale in every place review changed something: all four instances of
the artifact-loss defect, the gold-standard join that duplicates training
rows, the process pool that forks on Linux, the chunk size that serialises
the pool, a six-key metrics assertion where seven are reported, and code
that sorts nothing before two positional operations determining the model.

Anyone implementing from it would have reproduced the lot. Its job was to
drive the implementation and that is done. The design document stays and is
current. The plan survives outside the repo as the record of what was
planned, annotated with the nine places the shipped code deliberately
diverges from it.

The sort added in 2f155d5 is a total order only while `(studyLocusId, geneId)`
is unique. Polars' `sort` is not tie-stable, and `maintain_order=True` would
not close the gap, because it stabilises against the input order and the input
order is precisely what the joins in `derive_splits` leave unspecified. A
repeated pair would therefore put the tied rows back in an arbitrary relative
order and hand the positional draws below -- `subsample` and the SHAP
background -- back their run-to-run variation.

The failure would be invisible. `metrics.json` would still record its seeds,
`test_l2g_train_is_reproducible_across_runs` would still pass on its key-unique
fixture, and the only symptom would be a different model every release with
nothing to say why. `require_unique_key` now raises a `ValueError` after the
sort, naming the count and up to five of the offending pairs.

The new test feeds a gold standard carrying two curated rows that agree on
`(studyId, variantId, geneId)` and disagree on `goldStandardSet`, which is the
one way a pair survives `annotate`'s whole-row `unique` twice. The duplicate is
placed on a locus that survives the contamination anti-join, so it lands in the
train split rather than being filtered away first. Without the check the test
fails with `DID NOT RAISE ValueError`.

Follows the plan out. Both were working documents for the migration rather
than something this repository needs to carry: `docs/superpowers/` is a
tooling convention, and the repo has no other root-level docs tree.

The design survives outside the repo as the single current copy, with the
parity results and the corrections made during review. The PR description
carries what a reviewer needs to judge the change without it.

Measured on the real 26.09-2 inputs rather than estimated: 23m12s against
n1-highmem-32's 39m40s, a 1.71x improvement that matches a per-core
benchmark across five machine families. n2d and c3 cores are ~1.6x faster
than n1 at interventional TreeSHAP, which is 98% of this step.

Output is unchanged: the split statistics still reproduce all fourteen of
that release's recorded numbers, and output/l2g_prediction comes to the
same 564,562,707 bytes.

Keeps a memory-generous shape deliberately. Peak RSS is 28.2 GB, transient,
during the polars scan and join of the 5.5 GB of parquet inputs -- SHAP
itself is a flat ~12.7 GB. A 32 GB highcpu shape was tried and thrashed,
with under 4 GB left for the OS, page cache and the spawning workers. The
comment records this so the step is not "right-sized" into failing.